An example of a pen type input device with a camera, called a MEMO-PEN, is disclosed in ACM PRESS, HUMAN FACTORS IN COMPUTING SYSTEMS, CHI 95 Conference Companion, p256–P257. The MEMO-PEN incorporates a compact camera embedded in a penholder to continuously pick-up an image in the vicinity of a penpoint across the penpoint to record handwriting. The area in which data can be picked-up by the camera of the MEMO-PEN is limited to a quite narrow region required for making a judgment as to the direction of the handwriting.
In the MEMO-PEN, since the camera is embedded in the penholder, the visual field of the camera is obstructed by the hand of the user, unless the optical system is arranged at a position inclined toward the penpoint from a portion of the penholder to be gripped by the hand of the user. Therefore, even if an optical system having wide visual field is used, there is a limitation distance at which the optical system can be spaced from an object to be viewed (paper). Therefore, the wide visual field of the camera cannot be used efficiently. On the other hand, in the condition in which the user naturally holds the pen, the penholder is usually inclined significantly relative to an up-right position. Therefore, an image to be picked up by the camera is a cross shot of the image.
A function of the MEMO-PEN is to store the handwriting of the user written by the MEMO-PEN so as to allow the handwriting to be reproduced with the aid of an information processing apparatus or to recognize the written characters. Namely, during collection of the handwriting data, in otherwords, while the user uses the MEMO-PEN, a user interface, which is provided to call out the function of the information processing apparatus, using a pen-type input device with a camera, would never be executed. Accordingly, the kind of process to be applied for processing the object (handwriting) during input of data is not designated by the pen. Furthermore, in case of the MEMO-PEN, since the pen pointer and the position of the handwriting constantly match each other, it is not necessary to adjust the positional relationship between the input object (handwriting) and the penpoint.
As an example of a system relating to a pen-type input device with a camera, a system called PaperLink is disclosed in ACM PRESS, HUMAN-FACTORS IN COMPUTING SYSTEMS, CHI 97 CONFERENCE PROCEEDINGS, P327–p334. In the PaperLink system, the device with which the user inputs a process object is a pen-type input device mounting a compact camera. The compact camera is located at a position looking down on the object (paper surface) from right above under the condition in which the user naturally holds the pen. The visual field of the camera can be set so that the camera views an area of several cm square around the penpoint. The picked-up image around the penpoint is input to the information processing apparatus to execute various processes depending upon the data content. If the input object is known, a predetermined process depending thereupon is executed. For example, a predetermined file is opened to the user or execution of a predetermined program is initiated. On the other hand, if the object being input is not known, it is temporarily stored for use as an argument to a command to be executed later.
A problem to be solved by the present invention is that the usability of the device for inputting an image of an object is degraded by mounting the compact camera on the pen-type device. In the MEMO-PEN, since the camera including the optical system is built in the penholder, there is another problem in that it is difficult to provide a wide visual field for the camera.
Concerning the foregoing PaperLink system, since the axis of the pen and the center axis of the camera are arranged on the same plane, a longitudinally elongated pattern cannot be observed by natural operation. Furthermore, in the input device of the PaperLink system, since there is only one means for pointing to the object, it is not possible to designate the process object and kind of process simultaneously. Also, if the user designates an object and the penpoint has a construction such that the camera looks down the penpoint from above, as in the PaperLink system, the designated object is hidden by the penpoint, thereby to make it impossible to produce an accurate input.
On the other hand, since the penpoint and the designated object cannot be overlapped, as set forth above. It becomes necessary to limit the choice of the user in the method for designating the object. Namely, in the pen-type input device with a camera, the relationship between the position of the designated object and the position of the penpoint is variable depending upon the choice of designation method of the user. For example, when the user designates a laterally elongated object, some users may point to the center portion, and another user may point to a lower right position. On the other hand, the tilt angle of the pen upon pointing to the object may be different with different users.